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Abstract 

Background: Several systematic reviews and meta-analyses populate the literature on the effectiveness of 
laparoscopic surgery for colorectal cancer. The utility of this body of work is unclear. The objective of this study 
was to synthesize all such systematic reviews in terms of clinical effectiveness, to appraise their quality, and to 
determine whether areas of duplication exist across reviews. 

Methods: Systematic reviews comparing laparoscopic and open surgery for colorectal cancer were identified using 
a comprehensive search protocol (1991 to 2008). The primary outcome was overall survival. The methodological 
quality of reviews was appraised using the Assessment of Multiple Systematic Reviews (AMSTAR) instrument. 
Abstraction and quality appraisal was carried out by two independent reviewers. Reviews were synthesized, and 
outcomes were compared qualitatively. A citation analysis was carried out using simple matrices to assess the 
comprehensiveness of each review. 

Results: In total, 27 reviews were included; 13 reviews included only randomized controlled trials. Rectal cancer 
was addressed exclusively by four reviews. There was significant overlap between review purposes, populations 
and, outcomes. The mean AMSTAR score (out of 1 1) was 5.8 (95% CI: 4.6 to 7.0). Overall survival was evaluated by 
ten reviews, none of which found a significant difference. Three reviews provided a selective meta-analysis of time- 
to-event data. Previously published systematic reviews were poorly and highly selectively referenced (mean citation 
ratio 0.16, 95% CI: 0.093 to 0.22). Previously published trials were not comprehensively identified and cited (mean 
citation ratio 0.56, 95% CI: 0.46 to 0.65). 

Conclusions: Numerous overlapping systematic reviews of laparoscopic and open surgery for colorectal cancer 
exist in the literature. Despite variable methods and quality, survival outcomes are congruent across reviews. A 
duplication of research efforts appears to exist in the literature. Further systematic reviews or meta-analyses are 
unlikely to be justified without specifying a significantly different research objective. This works lends support to 
the registration and updating of systematic reviews. 
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Background 

Any field of active investigation in healthcare requires 
that the overwhelming volume of cumulative informa- 
tion generated by individual researchers be condensed 
and summarized into a usable product. This synthesis 
must be simple, yet comprehensive, so as to inform 
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decisions and policies carried out by physicians and sur- 
geons, hospital administrators, healthcare payers, fund- 
ing agencies, and other end users of research and 
outcomes data. Systematic reviews are widely considered 
to be the most comprehensive and unbiased method to 
do so [1]. On the basis of their completeness, such 
reviews should remain unique in the literature and be 
updated frequently, rather than duplicated or compart- 
mentalized. In this context, some authors have recently 
advocated for the open registration of systematic reviews 
[2-4]. 
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Laparoscopic colorectal surgery was first described in 
1991 by Fowler and White [5] and by Jacobs and collea- 
gues [6]. This technology has since been applied to 
almost every disease process, whether benign or malig- 
nant, involving the colon and rectum [7]. The use of 
laparoscopy rather than traditional open laparotomy to 
treat colorectal cancer has generated tremendous con- 
troversy in the surgical literature, particularly as con- 
cerns the oncologic adequacy of this technique. Many 
investigators have attempted to address this issue, and 
have thus generated a large body of literature over the 
past 20 years. Published studies have included the entire 
spectrum of research data, ranging from small personal 
case series to large nationally funded multicenter rando- 
mized controlled trials (RCTs). This work presents an 
excellent opportunity for a case study of research synth- 
esis and knowledge translation processes in surgical 
research, an area that has traditionally lacked investiga- 
tive rigor [8]. 

Numerous systematic reviews and meta-analyses per- 
taining to laparoscopic surgery for colorectal cancer 
have been published. Informal examination of these 
reviews would suggest significant overlap and possible 
duplication. The utility of this body of work is unclear 
at this time. As such, we set out to examine and 
appraise all existing systematic reviews of laparoscopic 
colorectal surgery for cancer, both in terms of clinical 
outcomes and their relative completeness, methodologi- 
cal quality, and overlap. 

Methods 

This overview of systematic reviews was carried out 
using the framework for umbrella reviews described by 
the Cochrane Collaboration [9]. This approach consisted 
of identifying all existing systematic reviews and meta- 
analyses pertaining to laparoscopic surgery for colorectal 
cancer. This work was part of a larger review effort 
addressing both primary publications and review papers, 
the results of which will be presented separately. A 
review protocol was utilized for the project as a whole. 

Criteria for considering reviews for inclusion 

All inclusion and exclusion criteria were defined a priori. 
All systematic reviews and meta-analyses addressing 
laparoscopic and open surgery for colorectal cancer were 
included. For this purpose, all reviews were allowable if 
they were self-described as systematic, whether in the title, 
abstract, or methods of the paper. Alternatively, a citation 
was also allowable if the authors presented a meta-analysis 
of primary papers or utilized meta-analytic techniques to 
pool primary data. These criteria were utilized regardless 
of the quality or comprehensiveness of the review. The 
type of primary data papers included in the citations could 
be RCTs, observational studies, or both. 



All included citations reviewed primary papers addres- 
sing the curative resection of colon and/or rectal cancer. 
Patients with colorectal cancer did not have to be the 
sole population under review. Laparoscopic resection for 
colorectal cancer was the intervention under study. 
Included reviews had to present a comparison to an 
open resection control group. 

The primary outcome of interest was overall survival. 
Secondary endpoints were also considered if they were 
included in a review of interest: operative outcomes, 
short-term postoperative outcomes, oncologic surrogate 
outcomes, long-term oncologic outcomes, other long- 
term outcomes, and quality of life. A review could be 
included in this overview even if no data pertaining to 
survival was presented, as identification of review deficits 
was a prespecified objective of our work. Reviews addres- 
sing exclusively cost or immune function were excluded. 

Search methods for identification of reviews 

Systematic reviews and meta-analyses were identified as 
part of a broader comprehensive search strategy designed 
to identify primary comparative literature pertaining to 
laparoscopic and open surgery for colorectal cancer. The 
final search algorithm was devised in conjunction with an 
experienced information specialist from the Ottawa Hos- 
pital Library (Additional file 1). This search strategy was 
designed to be highly sensitive, and was modified from 
previously published work [10]. Six major databases were 
searched for relevant citations from 1991 to 2008 (Ovid 
MEDLINE, Ovid EMBASE, Cochrane Library, Science 
Citation Index Expanded, BIOSIS Preview, and BIREME 
LILACS). An additional 13 databases were also searched 
for relevant citations (Database of Abstracts of Reviews 
of Effectiveness, Heath Technology Assessment Database, 
NHS Economic Evaluation Database, NIHR Health Tech- 
nology Assessment Programme, Trip Database, Clinical- 
trials. gov, Controlled-trials.com, National Guidelines 
Clearinghouse, CMA Infobase: Clinical Practice Guide- 
lines, NICE England, SIGN Scotland, NHMRC Australia, 
New Zealand Guidelines Group). The reference lists of 
all included citations were also screened to identify miss- 
ing reviews. No language limitation was applied to the 
search strategy. All citation records were retrieved and 
downloaded electronically using Reference Manager 10 
(ISI ResearchSoft, Berkeley, CA, USA), and were then de- 
duplicated. 

Selection of reviews 

All citations were first screened for inclusion by one 
reviewer (GM) on the basis of titles and abstracts (Fig- 
ure 1). All retained citations were then retrieved in full 
text. Papers that could not be obtained after extensive 
interlibrary searching were considered missing. Papers 
published in languages other than English, French, or 
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5,800 records after de-duplication 



5,709 records screened for inclusion 







920 full-text articles assessed for 
eligibility 






665 articles included in review 



421 review 
articles 



25 3 data 
articles* 



27 systematic 
review 



38 trials 



91 records missing 



4,789 records 
excluded: 

Animal studies 196 
Conference papers 77 
Diseese type 726 
Dupficate putfeation 28 
Irrelevant 2,077 
Intervention type 422 
Mated indications 372 
Outcome 110 
Purpose 153 
Pubfication dates 1 
Study type 627 



255 full-text 
articles excluded: 

Conference papers 50 
Disease type 2 
2-z :.d ::t:-:: 
Irrelevant 15 
Intervention type 10 
Mixed indications 47 
Outcome 42 
Purpose 39 
Study type 45 



215 observational 
studies** 



394 narrative 
reviews ** 



• 9 citations w ere included as data papers and narrative reviews 

Observational studies and narrative reviews w ere included in the review protocol, but 
they have been included i n a separate study. 

Figure 1 PRISMA diagram [95]. Note that the total number of data and review articles does not add up to 669, as 9 papers were included in 
both categories. 



Spanish were translated in full using Google Translate 
(Google Inc., Mountain View, CA, USA). Full-text arti- 
cles were evaluated for inclusion by one reviewer (GM). 
Included articles were then classified as (1) data papers 
or (2) review papers. Review papers were then further 
divided as systematic reviews/meta-analyses, narrative 
reviews, textbook chapters, and guidelines/position 
papers. Only systematic reviews/meta-analyses were 
considered in the current work, while all other included 
paper types were set aside for a separate research 



project. All included systematic reviews were further 
evaluated for inclusion by a second reviewer (SD), and 
disagreements between the two reviewers was resolved 
by discussion and consensus. 

Data extraction and management 

A total of 6 study characteristics and 19 prespecified 
outcomes of interests were extracted by two reviewers 
(GM, SD) onto data tables designed a priori (Table 1). 
All discrepancies were resolved by discussion and 
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Table 1 List of characteristics and outcomes extracted from each review 



Review characteristics Review outcomes 



Date assessed as up to date 


Positive radial margin (%) 


Postoperative mortality (%, 30 day) 


Population 


Positive distal margin (%) 


Length of hospital stay (days) 


Intervention(s) 


Radial margin distance (cm) 


Overall morbidity (%) 


Study types 


Distal margin distance (cm) 


Local recurrence (%) a 


Reported outcomes 


Operative time (min) 


Distal metastases (%) a 


Review limitations 


Nodes harvested (n) a 


Port site or wound metastases (%) a 




Anastomotic leak (%) 


Cancer-related mortality (%) a 




Pneumonia (%) 


Disease-free survival (%) a 




Urinary tract infection (%) 


Overall survival (%) a 




Surgical site infection (%) 





a Oncologic outcomes included in Additional file 2. 



consensus. Relative outcomes were always recorded as a 
comparison of laparoscopic to open surgery. Where 
papers compared open to laparoscopic surgery, the 
appropriate reciprocal relative measure of effect and 
reciprocal measure of significance were calculated and 
recorded. The relative measures of effect were main- 
tained in the statistical format provided by the authors 
of individual citations. 

Assessment of methodological quality of included 
reviews 

The methodological quality of individual reviews 
included in this overview was assessed using the Assess- 
ment of Multiple Systematic Reviews (AMSTAR) instru- 
ment [11]. This tool consists of 11 individual items, 
which were developed from pre-existing instruments, 
empirical evidence, and expert consensus. It has been 
shown to be valid, reliable, and relatively easy to use 
[11-13]. Each item within the instrument can receive 1 
point, for a possible range of AMSTAR scores of 0 to 
11. The AMSTAR instrument was administered inde- 
pendently by two reviewers (GM, SD) and discrepant 
scores were resolved by discussion and consensus. 

Data synthesis 

The data extracted from each included systematic 
review was incorporated into summary tables and fig- 
ures. The resulting data were synthesized in narrative 
form and correlated qualitatively with methodological 
AMSTAR scores. The congruence of pooled summary 
estimates for outcomes of interest across reviews was 
evaluated qualitatively. 

As a means of evaluating the appropriateness of 
included reviews, a bibliographic analysis was carried 
out. Citation matrices [14] were generated that cross- 
linked individual reviews with (1) previously published 
systematic reviews identified in this work, and (2) pub- 
lished RCTs of laparoscopic vs open colorectal cancer 
surgery also identified in this work. This technique was 



used to determine whether the authors of included 
reviews made use of other pre-existing systematic 
reviews, and whether the identification of relevant RCTs 
was comprehensive. A 1-year lag time to publication 
was incorporated into the citation matrices and calcula- 
tions. To account for the growing number of publica- 
tions in time, the ratio of cited to total pre-existing 
publications was calculated for each systematic review. 
The mean ratios and 95% confidence interval were cal- 
culated for citation of both pre-existing systematic 
reviews and RCTs. Finally, citation ratios for individual 
systematic reviews were compared to AMSTAR scores 
and 5-year impact factors (where available) [15] using 
Pearson correlation coefficients. 

All calculations and statistical analyses were per- 
formed using Microsoft Excel 2010 (Microsoft Corpora- 
tion, Richmond, WA, USA) and SAS V. 9.2 (SAS 
Institute Inc., Cary, NC, USA). 

Results 

Description of included reviews 

A total of 5,800 single citation records were screened for 
inclusion (Figure 1). After applying a 2-step screening 
process, a total of 27 full-text systematic reviews and/or 
meta-analyses [10,16-41] were included in this overview. 
Not included in this final tally was one Chinese lan- 
guage review, which could not be translated [42]. Simi- 
larly, another systematic review was excluded, as it 
focused solely on the methodological quality of RCTs 
identified in a Cochrane Review by the same author 
[43]. Finally, two more reviews were not considered as 
they addressed cost outcomes [44] and hand-assisted vs 
standard laparoscopic colorectal surgery [45], respec- 
tively. Other primary data papers and reviews identified 
in Figure 1 have also been included, and will form the 
basis for a separate publication. The full list of citations 
is available from the authors upon request. 

The characteristics of all included reviews are pro- 
vided in Table 2. Included reviews were published 
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Table 2 Characteristics of included reviews 



Review (origin) Last 

update 



Population/intervention 3 



Study types Notes/limitations 



Bernard/ANDEM, July 1993 

1994 [16], 

(France) 



Chapman/ 
ASERNIP-S 2000 
[17] (Australia) 

Vardulaki/NICE 
2000 [18], (UK) 



July 1999 



2000 



Chapman 2001 July 1999 
[19], (Australia) 



Yong 2001 [20], March 
(UK) 1997 



Korolija 2003 [21], January 
(Croatia) 2000 



Abraham 2004 December 

[22], (Australia) 2002 

Reza/UETS 2004 September 

[23], (Spain) 2004 

Poutignat/Haute 2003 
Autorite de Sante 
2005 [24], 
(France) 

Manterola 2005 December 

[25], (Chile) 2002 



Schwenk 2005 
[26], (Germany) 

Aziz 2006 [27], 
(UK) 



January 
2005 

2004 



Tjandra 2006 [28], September 

(Australia) 2005 

Reza 2006 [29], November 

(Spain) 2005 



Breukink 2006 August 
[30], ahe 2006 
Netherlands) 



Any colorectal pathology and other abdominal 
pathologies (appendicitis, biliary disease, reflux, 
inguinal hernia) 

Colon cancer (right, left/sigmoid) and animal 
studies of in vitro/in vivo tumor spread. 
Excluded: transverse colectomy, APR, total 
colectomy. 

Colorectal cancer. Excluded: case series of < 10 
patients. 



Colon cancer (right, left/sigmoid). Excluded: 
transverse colectomy, APR, total colectomy. 



Any colorectal pathology (all operation types) 



'Colorectal procedures', unclear 



Colorectal cancer (right hemicolectomy, left 
hemicolectomy, rectosigmoid resection, APR, 
others) 

Colorectal cancer. Excluded: combination of 
benign and malignant pathologies, immune 
outcomes. 

Colorectal cancer. Excluded: metastatic disease, 
combined benign and malignant pathologies, 
non-intention-to-treat studies or those excluding 
converted patients. 

Colon cancer (right, transverse, left, sigmoid). 
Excluded: rectal cancer, perforated or 
obstructing cancer, metastasis, recurrent cancer, 
invasion into adjacent bowel/organs, polyps. 

Colorectal pathologies (benign or malignant, for 
curative or palliative resection) 

Rectal cancer (described as anterior resection or 
APR) 



Case series; no 
RCT or 
observational 
study found 

RCTs, 

observational 
animal studies 

RCTs, 

observational 



RCTs, 

observational 



Observational; 
no RCT found 



RCTs, 

observational 



RCTs 



RCTs, systematic 
reviews, meta- 
analyses 

RCTs, 

observational, 
meta-analysis 



RCTs, 

observational 



RCTs 



RCTs, 

observational 



Colon and rectosigmoid cancer. Excluded: rectal RCTs 
cancer, distant metastases. 

Colorectal cancer. Excluded: papers with mixed RCTs, systematic 

malignant/benign populations, immune function reviews 
as outcome. 

Rectal cancer (undergoing total mesorectal RCTs, 

excision) observational 



In French. Paucity of available comparative 
literature. Broad review addressing all 
laparoscopic techniques (biliary, hernia, etc.). 

Limited to English. Inclusion/exclusion of high 
and mid rectal cancer is not clearly specified. 
Chose not to pool data. Overlap with 
Chapman 2001 [19]. 

Extensive methodological description. Rigorous 
assessment of heterogeneity. Pooling of RCTs 
and observational data performed separately to 
avoid bias (for some outcomes). Statistical 
manipulations to overcome data limitations. 

Limited to English. Inclusion/exclusion of high 
and mid rectal cancer is not clearly specified. 
Chose not to pool data. Overlap with 
Chapman 2000 [17]. 

Modification of inclusion/exclusion criteria from 
protocol based on available studies. Only 13/42 
studies had only malignant disease. Pooled 
certain outcomes by calculating weighted 
means. 

No inclusion/exclusion criteria provided. Not 
mentioned whether RCTs included (two 
identified in reference list). Chose to pool 
outcomes from RCTs, observational studies, 
case series together (no mention of 
appropriateness). 

Limited to English. Study selection on basis of 
reported outcomes. 

In Spanish. Pre-2000 RCTs not included. No 
pooling of outcomes except survival and 
recurrence (reason unclear). 

In French, limited to English and French. 
Unclear from text whether inclusion/exclusion 
criteria chosen a priori or after literature search 
by group of experts. Chose not to pool 
outcomes, qualitative analysis. 

In Spanish, limited to English, Spanish, French, 
Italian. Methodology score created by review 
authors. Controlled series and RCTs broken 
down into individual case series yielding 6 
laparoscopy and 1 1 open series in total. 

Cochrane Review, protocol published in 2001. 
Extensive methodological description. Results 
for different pathologies pooled together. 

Extensive quantitative assessment of 
heterogeneity. Results for RCTs and 
observational studies pooled (no sensitivity 
analysis comparing RCT to observational data). 

Limited to English. Effect of heterogeneity on 
results not clear/not documented. 

Pre-2000 RCTs not included. Limited 
description of methodology. No pooling of 
outcomes except survival and recurrence 
(reason unclear). 

Cochrane Review, protocol published in 2005. 
Primary outcome: disease-free survival. Chose 
not to pool survival data; qualitative analysis. 
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Table 2 Characteristics of included reviews (Continued) 



Gao 2006 [31], June 2005 
(China) 



Murray 2006 [10], May 2005 
(UK) 



Kahnamoui 2007 2004 
[32], (Canada) 

Noel 2007 [33], January 

(USA) 2005 



Bonjer 2007 [34], 2006 
(The Netherlands) 



Jackson 2007 
[35], (USA) 



February 
2006 



Abraham 2007 December 
[36], (Australia) 2003 



Kuhry 2007 [37], April 2006 
(Norway) 

Kuhry 2008 [38], January 

(Norway) 2008 



Lourenco 2008 
[39], (UK) 

Anderson 2008 
[40], (USA) 



May 2007 



November 
2007 



Liang 2008 [41], January 
(China) 2007 



Rectal cancer 



Colorectal cancer (including laparoscopic or 
HALS, excluding palliative surgery) 



RCTs, 

observational 



RCTs, IPD meta- 
analyses 



Colorectal cancer (right, left, sigmoid, anterior RCTs 
resection, APR) 



Colorectal cancer, IBD, diverticular disease 



RCTs, 

observational 
(controlled) 



Colon cancer (rectal cancer included in at least RCTs 
one of trials) 



Colorectal cancer (colon and rectosigmoid as 
per inclusion criteria, but selected RCTs include 
rectal cancers). Excluded: benign pathologies. 



RCTs 



Colorectal cancer (non-metastatic, treated with Observational 
intention to cure). Excluded: uncontrolled series. 



Colorectal cancer RCTs 

Colorectal cancer (reporting long-term result, RCTs 
non-metastasized carcinoma) 



Colorectal cancer. Excluded: patients undergoing RCTs, IPD meta- 
palliative treatment. analyses 



Rectal cancer. Excluded: tumors invading 
adjacent organs, previous pelvic surgery, 
contraindications to pneumoperitoneum, 
obstruction, perforation, studies which did not 
report rectal cancer outcomes separately. 

Colorectal cancer. Excluded: emergency surgery 
(obstruction, perforation), known prohibitive 
adhesions, studies for which colorectal cancer 
patients could not be analyzed separately from 
patients with benign pathologies. 



RCTs, 

observational 



RCTs 



Outcomes considered were selected post hoc 
after reviewing selected literature. Meta-analysis 
of RCT and observational data. Incomplete 
assessment of heterogeneity. 

HTA report (commissioned), protocol published 
in 2005. Pre-2000 RCTs identified from existing 
systematic reviews. Extensive description of 
methodology, rigorous assessment of 
heterogeneity. Academic-in-confidence data 
obtained from other authors removed from 
final report. Includes economic evaluation. 

Defined primary outcome: cancer-related 
mortality. Extensive methodological description. 
Quality appraisal list designed by authors. 

Limited to English. Excluded historical controls 
with < 50% overlap in accrual periods. 
Combination of RCT and non-RCT data. No 
assessment of methodological quality. 

Minimum 150 patients with primary outcomes 
of survival. Authors of review are primary 
investigators in four included trials. Meta- 
analysis of individual patient data with 3 years 
of follow-up data. 

Primary outcome: survival and recurrence. 
Inclusion/exclusion of mid and low rectal 
cancer is not clearly specified. Meta-analysis of 
survival outcomes using time-to-event data. 
Significant assessment of heterogeneity. 

Limited to English. Limited to end of 2003. 
Quality assessment of papers performed but 
not utilized in analysis (to be reported 
separately). Limited assessment of 
heterogeneity. 

In Norwegian, limited to English. Very limited 
methodological description. 

Cochrane review, protocol published 2002. 
Meta-analysis of survival outcomes using time- 
to-event data. Sensitivity analyses performed 
separately for colon and rectum. 

Pre-2000 RCTs identified from existing 
systematic reviews. Extensive description of 
methodology. Overlap with Murray 2006 [10]. 

Limited to English language. RCT and 
observational studies pooled together. 



Limited to English. Extensive assessment of 
heterogeneity. 



"laparoscopic versus open radical oncologic resection is the intervention under study unless stated otherwise. 

APR = abdominoperineal resection; HALS = hand-assisted laparoscopic surgery; HTA = Health Technology Assessment; IBD = 

individual patient data; RCT = randomized controlled trials. 



inflammatory bowel disease; IPD = 



between 1994 and 2008, of which 19/27 (70%) were 
published in 2005 or later. Three groups published 
updates of their own reviews [10,17,19,23,29,39]. 
Because these were not labeled as updates in the titles 
or abstracts, they were included as independent reviews. 
A majority of reviews (n = 18, 67%) were published in 
traditional print journals, while the remainder 



comprised commissioned reports (n = 4, 15%) 
[16,17,23,24], Cochrane Reviews (n = 3, 11%) [26,30,38], 
and Health Technology Assessment (HTA) report (n = 
2, 7.4%) [10,18]. The mean 5-year impact factor asso- 
ciated with published reports was 4.02 (n = 20, 95% CI: 
3.07 to 4.96). These originated in 12 countries, with 
Europe (n = 14, 52%) and Australia (n = 5, 19%) 
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accounting for a majority of papers. Five reviews were 
presented in non-English languages, including French (n 
= 2), Spanish (n = 2), and Norwegian (n = 1). 

Among included reviews, four (15%) addressed exclu- 
sively rectal cancer [27,30,31,41], one (3.7%) reviewed 
only colon cancer [25], whereas the remainder were less 
specific and identified colorectal cancer' as their popula- 
tion of choice (Tables 2 and 3). This descriptor led to 
contradictions in certain papers, as the authors some- 
times specifically excluded rectal cancer, but then went 
on to include papers with 'rectosigmoid cancers' or 
'anterior resections' [17,19,34]. A total of four (15%) 
reviews were wide in scope and addressed the laparo- 
scopic treatment of all colorectal pathologies, of which 
colorectal cancer was a subset of patients [16,20,21,33]. 
All reviews sought to find studies comparing laparo- 
scopic and open radical resection for cancer. One early 
paper found no controlled study, and was thus limited 
to case series of laparoscopic surgery for colorectal can- 
cer [16]. Right hemicolectomy, left hemicolectomy, and 
sigmoid resection were the most commonly included 
types of colonic resections. All publications not limited 
exclusively to rectal cancer included those three proce- 
dures. In contrast, studies including transverse colec- 
tomies were frequently excluded by review authors 
[17,19,22,32]. 

All reviews except one set out to include RCTs as part 
of their analyses (Table 2). One group chose to focus 
solely upon observational studies [36]. A total of 13 
(48%) reviews allowed only RCTs as part of their inclu- 
sion criteria. Of these, 77% were published in 2006 or 
later. One such group selected only four larger RCTs, 
and carried out a meta-analysis of individual patient 
data [34]. Two reviews whose last literature searches 
were in 1993 [16] and 1997 [20], respectively, were 
unable to identify any published RCTs. As a result, both 
groups presented only observational studies. 

Table 3 provides an overview of the purposes, disease 
processes, and outcomes addressed by each systematic 
review. The self-described purposes were highly compar- 
able, with most papers choosing to address issues of effi- 
cacy or effectiveness in broad terms. Long-term 
oncologic outcomes were sought by 19/27 reviewers 
(70%), while oncologic surrogates were found in 22 (81%) 
reviews. Similarly, operative and short-term postoperative 
outcomes were analyzed in 74% (n = 20) and 78% (n = 
21) of reviews, respectively. A total of 13 (48%) reviews 
addressed all 4 outcomes types, while 18 (67%) included 
at least 3 of the 4. Table 3 reveals significant overlap in 
study purposes and outcomes. In contrast, long-term 
operative outcomes (n = 4, 15%) and perioperative qual- 
ity of life (n = 6, 22%) were much less frequently included 
in the current group of systematic reviews. 



Methodological quality 

The included systematic reviews were generally of low 
to moderate quality. The mean AMSTAR methodologi- 
cal quality score was 5.8 (95% CI: 4.6 to 7.0). A total of 
eight reviews (30%) achieved a score of 9 or greater, and 
can be considered of high methodological quality. The 
quality of reviews appears to have improved modestly in 
time (Figure 2), with all but one high-quality review 
having been published in 2005 or later. Among high- 
quality reviews, three were published in traditional jour- 
nals [32,35,39], three were Cochrane Reviews [26,30,38], 
and two were HTA reports [10,18]. 

The composite of individual items within the 
AMSTAR score is presented in Figure 3. Most review 
groups defined their research question and inclusion 
criteria a priori {20127, 74%), and documented the eva- 
luation of the scientific quality of included studies (20/ 
27, 74%). Similarly, 70% of systematic reviews used 
appropriate methods to combine individual study 
results (n = 19), and can be considered to have 
reached appropriate conclusions that reflected the 
methodological quality assessment of their selected 
papers (n = 19). In contrast, few groups addressed the 
possibility of publication bias (7/27, 26%), provided 
evidence of a duplicate study selection and data extrac- 
tion process (10/27, 37%), listed all included and 
excluded studies (10/27, 37%), or avoided limiting their 
search or inclusion of studies on the basis of publica- 
tion status (9/27, 33%). 

Synthesis of reviewed outcomes 

The primary outcome of overall survival was pooled 
quantitatively in ten reviews (Figure 4 and Additional 
file 2). Most authors presented this data as a dichoto- 
mous outcome (alive/dead) at maximum follow-up. 
After meta-analysis, this data was presented as odds 
ratios (OR), risk ratios (RR), and Stouffer's composite Z 
(Z c ). Two groups presented this outcome as time-to- 
event data by pooling hazard ratios (HR), using methods 
relying on the estimation of HR from Kaplan-Meier 
curves. Finally, another group also presented a HR that 
was derived from an individual patient data meta-analy- 
sis. No significant difference in overall survival was 
found between laparoscopic and open surgery for color- 
ectal cancer across all meta-analytic comparisons. The 
direction of effect for all analyses favored laparoscopy, 
except for two which were conducted on observational 
studies and using a subset of studies with independent 
patient data [18,34]. 

All other pooled outcomes, except for the total num- 
ber of harvested lymph nodes, yielded comparable non- 
significant results. These findings are not presented in 
the main text, but are available in Additional file 2. 
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Table 3 Summary of reviewed populations and outcomes 







Disease process 




Outcomes reviewed 










Review 


Purpose 


Colon 
cancer 


Rectal 
cancer 


Other 


Operative 


Short-term 
postoperative 


Oncologic 
surrogate 


Long-term 
oncologic 


Long- 
term 
other 


QoL 


Bernard 
1994 [16] 


Safety, efficacy 


Y 


Y 


Y 


Y 


Y 


Y 


Y 






Chapman 
2000 [1 7] 


Safety, efficacy 


Y 


X 




Y 


Y 


Y 


Y 






Vardulaki 
2000 [18] 


Effectiveness, cost 
effectiveness 


Y 


Y 




Y 


Y 


Y 


Y 






Chapman 
2001 [19] 


Safety, efficacy 


Y 


X 




Y 


Y 


Y 


Y 






Yong 2001 
[20] 


Effectiveness 


Y 


Y 


Y 


Y 


Y 


Y 


Y 






Korolija 2003 
[21] 


Extent of oncologic 
resection 


Y 


Y 


Y 






Y 








Abraham 
2004 [22] 


Safety, efficacy (short 
term) 


Y 


Y 




Y 


Y 


Y 








Reza 2004 
[23] 


Safety, efficacy 


Y 


Y 




Y 


Y 


Y 


Y 




Y 


Poutignat 
2005 [24] 


Safety, efficacy 


Y 


Y 




Y 


Y 


Y 


Y 




Y 


Manterola 
2005 [25] 


Identify best therapeutic 
option 


Y 








Y 




Y 






Schwenk 
2005 [26] 


Short-term benefits 


Y 


Y 




Y 


Y 








Y 


Aziz 2006 
[27] 


Short/long-term results 
(rectal) 




Y 




Y 


Y 


Y 




Y 




Tjandra 2006 
[28] 


Update short-term results 


Y 


Y 




Y 


Y 


Y 








Reza 2006 
[29] 


Safety, efficacy 


Y 


Y 




Y 


Y 


Y 


Y 






Breukink 
2006 [30] 


Safety, efficacy (rectal) 




Y 




Y 


Y 


Y 


Y 




Y 


Gao 2006 
[31] 


Safety, efficacy, benefits 
(rectal) 




Y 




Y 


Y 


Y 








Murray 2006 
[10] 


Effectiveness, cost 
effectiveness 


Y 


Y 




Y 


Y 


Y 


Y 


Y 


Y 


Kahnamoui 
2007 [32] 


Non-inferiority survival/ 
perioperative results 


Y 


Y 




Y 


Y 


Y 


Y 






Noel 2007 
[33] 


Safety, efficacy (short 
term) 


Y 


Y 


Y 


Y 


Y 










Bonjer 2007 
[34] 


Safety (oncologic) 


Y 


X 








Y 


Y 






Jackson 
2007 [35] 


Compare oncologic 
results 


Y 


Y 








Y 


Y 






Abraham 
2007 [36] 


Safety, efficacy (non-RCT, 
short term) 


Y 


Y 




Y 


Y 


Y 








K\ ihrv 1007 
i\ui ii y zuu/ 

[37] 


InUL bldlcU 


Y 


Y 




Y 


Y 


y 


y 






Kuhry 2008 
[38] 


Evaluate long-term 
outcomes 


Y 


Y 










Y 


Y 




Lourenco 
2008 [39] 


Update, effectiveness 


Y 


Y 




Y 


Y 


Y 


Y 


Y 


Y 


Anderson 
2008 [40] 


Compare oncologic 
outcomes (rectal) 




Y 








Y 


Y 






Liang 2008 
[41] 


Evaluate recurrence 
outcomes 


Y 


Y 










Y 







QoL = quality of life; x = Unclear from descriptions whether includes high rectal cancers. 



Martel et al. Systematic Reviews 2012, 1:14 
http://www.systematicreviewsjournal.eom/content/1 /I /1 4 



Page 9 of 1 7 



12 



1C 



2 

s 











• < 


♦ 

I 








• 






> ♦ » 












♦ — 

♦ 


♦ 








♦ 


# 


♦ ♦ 






♦ 








♦ 

♦ 


♦ ♦ 










♦ 


♦ 

A 


♦ 





Jan 1993 



Oct 1995 



Jul 1998 



Apr 2001 Jan 2004 

Time 



Oct 2CC6 



Ju 2CC9 



Apr 2C12 



Figure 2 Assessment of Multiple Systematic Reviews (AMSTAR) methodological quality scores against time. Red circles represent 
Cochrane reviews and Health Technology Assessment (HTA) reports. 



Bibliographic analysis 

The pattern of citations of previously published systema- 
tic reviews is presented in Figure 5. In total, nine (33%) 
reviews did not cite any pre-existing work. The maxi- 
mum number of cited existing systematic reviews was 
five, which was found in two reviews [29,40]. All other 
studies cited zero to four pre-existing systematic 
reviews. The mean number of cited existing reviews was 
1.6 (95% CI: 1.0 to 2.2). The mean ratio of cited to total 
existing systematic reviews was 0.16 (95% CI: 0.093 to 
0.22), ranging from 0 to 0.5. All three Cochrane Reviews 
(0, 0.048 to 0.091) and the more recent HTA report 
(0.18) had citation ratios that were less than 0.2. There 
was no correlation between the citation ratio and the 
AMSTAR score (r = 0.047) or the journal's 5-year 
impact factor (r = -0.099) for individual reviews. In 
total, 13/26 pre-existing reviews were cited at least once, 
with 5 of these accounting for 71% of all citations (29/ 
41) [17-19,22,26]. Figure 5 shows that five reviews were 
cited disproportionately more frequently, and that all 



five reviews were published in the earlier portion of the 
literature review. 

The current overview protocol identified 38 publica- 
tions pertaining to 23 individual RCTs comparing 
laparoscopic and open surgery for colorectal cancer 
[46-83]. The patterns of citations of these RCTs are pre- 
sented in Figure 6. Two systematic reviews did not cite 
any RCTs. In the case of Bernard et al. [16], there were 
no RCTs yet published in the literature. In the case of 
Yong et al. [20], a total of ten RCT publications could 
have been cited. The mean ratio of cited to total pub- 
lished RCT reports was 0.45 (95% CI: 0.35 to 0.54), ran- 
ging from 0 to 1. Excluding Abraham et al. [36], which 
sought to analyze only observational studies, yielded a 
comparable mean ratio of 0.46 (95% CI: 0.36 to 0.56). 
Given that certain RCTs were published over multiple 
papers, the ratio of cited to total published RCT reports 
was recalculated using each trial as the denominator 
rather than individual papers. This analysis yielded a 
mean ratio of 0.56 (95% CI: 0.46, 0.65). A total of only 
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four reviews identified at least 75% of RCT publications, 
of which two were Cochrane Reviews and one was an 
HTA report [10,26,28,38]. Reviews that selected rectal 
cancer as their sole patient population had generally low 
citation ratios, ranging from 0.13 to 0.39 [27,30,31,40]. 
Among all RCT publications, two were cited dispropor- 
tionately more frequently than others. Indeed, the Bar- 
celona trial by Lacy et al. [57] and the Clinical 
Outcomes of Surgical Therapy (COST) trial [63] were 
both cited by 90% of systematic reviews. Finally, the cor- 
relation between the citation ratios and the AMSTAR 
scores (r = 0.43), and between the ratios and the jour- 
nal's 5-year impact factors were moderate at best (r = 
0.46). 

Discussion 

In this work, we have presented an overview of pub- 
lished systematic reviews and meta-analyses comparing 
laparoscopic and open surgery for colorectal cancer. We 
have conducted an extensive review of the literature 
using a highly sensitive search algorithm, and utilized 
the framework proposed by the Cochrane Collaboration 
[9] to synthesize 27 systematic reviews and meta-ana- 
lyses published between 1994 and 2008. We have sum- 
marized the major oncologic outcome of overall 
survival, and provided a methodological appraisal of the 
reviews. Finally, we have analyzed the citation patterns 



of all included reviews in an attempt to understand the 
perceived redundancy of this body of work. 

The first major finding of this overview pertains to the 
startling number of systematic reviews and meta-ana- 
lyses identified in the literature on laparoscopic colorec- 
tal cancer surgery. There were more reviews than 
clinical trials, and 19 reviews were published within the 
span of 4 years (2005 to 2008). Given that systematic 
reviews are typically meant to be comprehensive in nat- 
ure, this result is particularly surprising. This finding 
can be explained in part by the decision of certain 
reviewers to address only a subset of outcomes. That 
being said, the results shown in Table 3 would suggest 
that there is significant overlap between papers in terms 
of review purpose and outcomes. This argument is 
further supported by the large number of pooled ana- 
lyses identified for the primary survival outcome in Fig- 
ure 4 and in Additional file 2 in addition to the 
consistency and congruence of this result across reviews. 

In addition to variation in outcomes, other authors 
have limited their review protocols with respect to dis- 
ease subsets such as rectal cancer. This distinction may 
also account for a portion of the large volume of pub- 
lished systematic reviews, although it is unlikely to be a 
major factor as only four groups have focused solely on 
this population. More importantly, we argue that the 
limitation of review protocols to rectal cancer is 
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Reviews 



Vardulaki 2000 [IS) 



Reza2D04 [23] 



Murray 2006 [10] 
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Open / Lap studies 



s 



Bonjer 2007 [34] 



Jackson 2007 [35] 



Kuhry 200S [38] 



Lourenco 2008 [39] 



Anderson 2008 [40] 



76% (718) 



79% (827) 



76% (718) 



Kahnamoui 2007 [32] 76% (586) 



33,5% 



79% (827) 



65% 



79% (733) 



81% (843) 



79% (733) 



30% (508) 



82,2% 



74.5% (1309) 763% (1572) 



81% (843) 



72% 
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OR 
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OR 



OR 



HR* 



HR* 



OR 
HR* 



RR 



-+ 



♦ 



Cohen's d =0.1 



- Hazard ratios <1 favor laparoscopic surgery, as the hazard function compares the risk of death for each study group. 
Figure 4 Synthesis of overall survival across systematic reviews. 



artificial, as definitions and inclusion of rectal cancer in 
primary trials are highly variable. Indeed, high rectal 
lesions have been inconsistently defined both as recto- 
sigmoid and rectal cancers, and this variation in termi- 
nology has been translated to different inclusion and 
exclusion criteria in systematic reviews. Given that high 
rectal cancers are typically treated in a manner that is 
more similar to sigmoid colon cancers, we argue that 
rectal cancer is best considered as part of broader 
reviews addressing colorectal cancer as a whole. Sensi- 
tivity analyses can then be carried out to identify out- 
come differences between colon and rectal cancer 
populations. 

In addition to the above-mentioned patient population 
and outcomes factors, it is possible that the large num- 
ber of systematic reviews be a result of a duplication of 
research efforts on the part of individual investigators. 



The analysis of citation patterns of pre-existing systema- 
tic reviews presented in Figure 5 supports this hypoth- 
esis. Indeed, we have identified that, on average, only 
1.6 pre-existing reviews were referenced by individual 
systematic reviews. More strikingly, this corresponds to 
a mean ratio of cited to existing reviews of 0.16, indicat- 
ing that, on average, 84% of existing systematic reviews 
in the literature were not found or ignored by individual 
review authors. This finding would support the argu- 
ment that review authors either carry out poor literature 
evaluations or choose not to take into account pre-exist- 
ing work in performing or analyzing their own reviews. 
Individual explanations are likely to vary, but it remains 
that our findings provide evidence of a duplication of 
research efforts in the literature. 

Other examples of overlapping systematic reviews and 
meta-analyses can be found in the literature. In one 
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Figure 5 Citation of previously published systematic reviews 
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such case, ten reviews pertaining to the use of Af-acetyl- 
cysteine for the prevention of contrast-associated 
nephropathy were published within a 3-year span [84]. 
In that particular case study, the authors documented 
varying quality and inconsistent recommendations. 



Another such analysis was published in the field of 
orthopedic surgery, in which different methods of ante- 
rior cruciate ligament repair were compared [85]. A 
total of 11 overlapping systematic reviews were identi- 
fied in this case, with varying conclusions. Interestingly, 
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this group also identified evidence of incomplete citation 
of pre-existing systematic reviews. 

Several academics have recently called for the registra- 
tion of systematic reviews and their protocols [3,4]. 
Although the Cochrane Library currently acts as a cen- 
tral repository of high-quality systematic reviews and 
meta-analyses, it is clear from our analysis that most 
reviewers publish their work in traditional print journals. 
This interpretation is supported by Moher and collea- 
gues, who reported that approximately 2,500 new sys- 
tematic reviews are published annually, of which over 
90% are found in specialty journals [2]. As such, a freely 
accessible registration system would have several advan- 
tages, including the early identification of pre-existing, 
ongoing, or unpublished reviews, the prioritization of 
research funding, and the enhancement of collaboration 
between review groups, while minimizing the possibility 
of research duplication. This body of information would 
be of particular use to practicing surgeons who would 
now have access to systematic reviews and meta-ana- 
lyses through a single portal. Duplication of systematic 
reviews may be minimized with the existence of such an 
open-access registry, and identification of pre-existing 
work would likely be enhanced. 

The continuous publication of new trials in time leads 
to the production and dissemination of new systematic 
reviews as a means to provide a synthesis of the litera- 
ture that relies upon the latest data. For systematic 
reviews to be considered useful for end users, they must 
be up to date. In the current study, at least three groups 
of reviewers have provided updates of their own work in 
subsequent publications [10,17,19,23,29,39]. However, 
these were not specifically labeled as such, a finding that 
may lead to confusion on the part of the reader. In con- 
trast, all three Cochrane Reviews documented having 
undergone substantive amendments since their original 
publication, as a result of new data pertinent data in the 
literature [26,30,38]. This difference between Cochrane 
and non-Cochrane reviews is not surprising. Indeed, 
Jadad et al. have previously demonstrated that only 3% 
of systematic reviews published in traditional journals 
underwent update within 2 years of publication, com- 
pared with 38% of Cochrane Reviews [86]. Although the 
timing at which systematic reviews should be updated 
remains controversial, it seems intuitive that, in a 
rapidly progressing field of healthcare such as laparo- 
scopic surgery for colorectal cancer, existing systematic 
reviews should be updated frequently [87,88]. The case 
for updating systematic reviews becomes particularly 
compelling when one considers the large number of 
overlapping reviews identified in this study, and when 
registration of systematic reviews is considered. 

The methodological quality of systematic reviews 
included in this study was low to moderate. As 



indicated, the mean AMSTAR score was 5.6, although 
8/27 reviews achieved a score of >9. It is noteworthy 
that all Cochrane reviews and HTA reports in this study 
were classified as high-quality reviews. This result is 
supported by existing work in the literature, which 
demonstrated that Cochrane reviews have greater meth- 
odological rigor than traditional print reviews [86]. 

Although many systematic reviews in this work were 
deemed to have a comprehensive literature search (n = 
16, 59%) on AMSTAR methodology scoring, few incor- 
porated the gray literature (n = 9, 33%). As a result, 
many reports fell short in their identification of pub- 
lished trials. Indeed, the mean ratio of cited to existing 
RCT reports was only 0.46 across reviews. It was only 
marginally better at 0.56 if one considered actual trials 
rather individual reports of trials which were sometimes 
multiple (38 reports on 23 RCTs). Only one HTA report 
[10] managed to identify all reports of existing trials. 
This finding is concerning in that well conducted sys- 
tematic reviews are meant to be comprehensive and 
commonly believed to provide the highest degree of 
healthcare evidence [4]. 

Many factors may explain the finding of incomplete 
citation, including the differing patient populations and 
outcomes examined in each review. This is particularly 
relevant given the recent report by the ORBIT group on 
outcome reporting bias and its potential impact upon 
results of systematic reviews [89]. As highlighted in the 
ORBIT study, it is highly important that systematic 
reviews avoid excluding trials on the basis of a lack of 
relevant data, as authors may have elected not to report 
a given outcome. In addition, it is also likely that the 
time between systematic review search completion and 
dissemination in electronic or print format may account 
for a portion of the poor citation of trials. We have 
attempted to avoid this form of time lag bias by allow- 
ing for a reasonable 1-year gray zone between citation 
of RCTs and publication (Figures 5 and 6). However, 
several systematic reviews had time lags to publication 
in excess of 2 years [19-21,25,32,36] (Table 2), which 
may partially account for a failure to identify more 
recent RCTs. In addition to the above factors, it is also 
likely that individual search strategies were not as com- 
prehensive as should be, in order to identify all available 
trials. In the current study, we have modified the highly 
sensitive search algorithm developed by Murray and col- 
leagues [10] and have identified at least two recent trials 
that have not been cited in of the existing reviews 
[71,73]. Moreover, at least four other reports of RCTs 
were cited much less frequently than other reports of 
trials published around the same time period 
[49,54,59,68,70]. In contrast, two well known RCTs were 
cited by almost all reviews [57,63]. Putting together the 
above patterns of trials citations, we argue that the 



Martel et al. Systematic Reviews 2012, 1:14 
http://www.systematicreviewsjournal.eom/content/1 /I /1 4 



Page 14 of 17 



identification of trial evidence was incomplete in most 
retrieved systematic reviews, due at least in part to 
inadequate search strategies. 

Our overview of all systematic reviews presenting data 
on oncologic outcomes reveals several important find- 
ings. First, we found no evidence of a consistent or con- 
gruent difference in overall survival between patients 
with colon cancer treated by laparoscopy or open sur- 
gery. Similar conclusions can be drawn from other 
oncologic outcomes (data not shown; Additional file 2). 
This result is likely to be extendable to patients with 
high rectal cancer as many large trials included this 
pathology [62,66,76]. However, our analysis cannot be 
extrapolated to those with mid or low rectal cancer, as 
too few trials have included these patients. Results from 
the large multicenter and multinational COLOR II [90], 
ACOSOG Z6051 [91] trials will shed light onto this area 
of uncertainty. That being said, it is important to note 
that the above conclusion is limited by the lack of 
proper analysis of time-to-event data. Indeed, only three 
meta-analyses addressing overall survival presented their 
data in the form of HR [34,35,38]. Instead, many 
authors simply used pooled OR, which incorporate the 
proportion of patients alive or dead at a given point in 
time in each study. While this approach provides some 
information on survival, it is potentially biased by vari- 
able lengths of follow-up, different trial maturity, and 
the incomplete utilization of available data from patient 
censoring [38]. Because many RCT authors do not 
report HR, statistical methods exist to generate such 
estimates from Kaplan-Meier curves [92-94]. We advo- 
cate that review groups should attempt to gather this 
type of data when addressing survival or other oncologic 
outcomes. 

Conclusions 

A large number of overlapping systematic reviews and 
meta-analyses comparing laparoscopic and open surgery 
for colorectal cancer can be identified in the literature. 
The methodological quality of systematic reviews is gen- 
erally low to moderate, as evidenced by the incomplete 
identification of published trials. On the whole, 
Cochrane Reviews and Health Technology Assessment 
reports demonstrate higher quality indices than most 
traditional print reviews. Survival outcomes are inconsis- 
tently reported and time-to-event data are infrequently 
included in pooled estimates. That being said, all pooled 
estimates of overall survival comparing laparoscopic and 
open surgery for colorectal cancer are congruent and 
demonstrate no significant difference. There appears to 
be evidence of duplication of research efforts among 
review groups, as evidenced by overlapping review pur- 
poses, populations, and outcomes, as well as by the poor 
citation of pre-existing systematic reviews. Further 



systematic reviews or meta-analyses are unlikely to be 
justified without specifying a significantly different 
research objective. This works lends support to the 
registration and updating of systematic reviews. 

Additional material 



Additional file 1: Ovid MEDLINE (1950 to July Week 4 2008) 
Additional file 2: Summary of oncologic outcomes. 
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